Confusion with Deduplicating; No Longer Creating Addtl Alerts on the Same Issue

#1

Something has changed recently thatā€™s broken our use of PagerDuty and I donā€™t know how to fix it.

Previously, weā€™d generate alerts via e-mail and deduplicate them based on the Ticket number our system generates in the subject line. Anything between # and / would belong to the same incident/alert.

(FYI - Iā€™m using those two words, alert and incident, interchangeably as weā€™ve been with PagerDuty since before these were separate things. Iā€™m still not totally sure I understand what the difference is between these two things as it applies to our use of PagerDuty and this might be the source of our problems.)

So, as I was saying any e-mail with the same ticket number in the subject line would be deduplicated. This is because our system will send out multiple e-mail alerts about the same ticket every 10 minutes until the issue was addressed in our system. This allowed PagerDuty to only have one incident/alert in the system and generate only one page.

Once the issue was addressed in our system, an e-mail would kick off to PagerDuty closing out the incident/alert and it would be shown as resolved in the PD system.

If in the future however, the same ticket number would enter a failed state and starting generating alerts again, a new incident/alert would be created in PagerDuty then send off the page. Again, additional messages from our system about the same issue would be deduplicated until it was addressed in our system and the ā€œall clearā€ was sent to PagerDuty to close out the incident/alert.

The problem is that second thing is no longer happening.

Once a ticket generates an alert/incident in PagerDuty and it gets resolved, if that same ticket goes back into a failed state again at a future date and the e-mail with the alert gets sent off to PagerDuty, PagerDuty does nothing with that e-mail. It doesnā€™t generate another incident/alert and we donā€™t get paged.

This is obviously horrible as this means weā€™re missing important pages.

Iā€™m not sure when this problem started happening. As pages were still coming in it seemed like it was working, but at least once or twice something that should have generated a page didnā€™t generate one and I just figured it might be a fluke. The same thing happened a third time this morning so I started doing testing and figured out the pattern mentioned above.

Anyone have any idea what we might need to change to work inside this new incident/alert system? Iā€™m assuming thatā€™s somehow related as to why this is no longer triggering a new alert/pageā€¦ but I could be wrong. Iā€™m just guessing and grabbing at straws here.

Thanks for any help or ideas you might have.

ā€“ Justin Swall

#2

A few questions to get some more context:

  • Are you using an event orchestration or event ruleset email integration or a service level email integration?
  • Are you using regex to extract the ticket ID from your email subject and use that for your dedup key or are you just using the entire subject line as-is?
  • Is the subject line changing at all from the first trigger email and follow on emails? (for example, do follow on emails have ā€œRe:ā€ or ā€œFwd:ā€ or anything different? (assuming youā€™re using full subject line here)
  • Are you defining certain conditions to map into a unique 'trigger" and ā€˜resolveā€™ action in your rules configuration?
#3

Thanks for the reply @dmcclure

Our setup is very simple since itā€™s been humming along for years before things like Event Orchestration were part of the platform. Essentially, we just have a couple of Services defined. Each service has only three settings enabled - the On Call Now, the Escalation Policy, and an Email Integration.

Each Email Integration is configured identically to the one shown below. The only reason we even have multiple services defined is just because then it shows the service impacted on the notification in the Pager Duty app. We can then route the various types of alerts to the various e-mail addresses tied to the various services and get a bit more clarity on what type of an alert is coming in.

image
image.png814Ɨ453 45.7 KB

Let me know if thereā€™s any additional information I can provide. Thanks!

#4

Are dedup/incident keys getting created as expected with the value between # and / ? If so, can you manually send in an email that should trigger and one that should resolve and have the incident created and resolved as expected?

I might also try creating a new service and email integration and test with manually sending an email just to sanity check if it works or not with a new service/email address.

#5

The plot thickens.

Iā€™ve done some additional testing this evening and when I do my manual test it is all working correctly as expected. I do still have a specific e-mail that I can confirm was sent that never kicked off a page, so Iā€™ve reported the issue to support.

Doesnā€™t seem like this will be a forum-solvable problem. Thanks for all of your help though!

1 Like